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Abstract 

Minimization plays a central role in science and technology, which has motivated much related attention and efforts, 
giving rise to several interesting methods and algorithms. In this work, we provide an introduction to some basic 
concepts and approaches to minimization of functions. We start with a simple random search, deriving some interesting 
insights about the challenges in minimization, including the steeply increasing computational costs implied by higher 
dimensions. Then, we describe a zooming version of the random search, which can allow enhanced performance in some 
situations. The local approach known as gradient descent is also briefly discussed, identifying some of its advantages and 
disadvantages. The statistic approach known as simulated annealing is also briefly introduced. These four minimization 
methods are illustrated with respect to a case example involving local minimum. 


‘La sapienza e figlia dell’esperienza.’ 

Leonardo da Vinci. 

1 Introduction 

In science and technology, we are often interested in find¬ 
ing extreme, i.e. maximum or minimum, values of some 
measurement or property implied by some real-world or 
abstract system. For instance, knowing the maximum and 
minimum value of a function allows us to normalize it so 
as to be best visualized or treated numerically (optimizing 
the numeric resolution). Or, we can be interested in find¬ 
ing the minimal cost or error of some problem expressed 
in terms of some respective function. Though sometimes 
the scalar field equation is given, in many other situa¬ 
tions we can only sample values from the function to be 
minimized. 

Training neuronal networks represents one important 
minimization task, where the error can be understood 
as being proportional to the difference between the cur¬ 
rent neuronal network output and the sought answer for 
a given input, and the weights of the neurons, understood 
as the optimization domain, have to be adjusted so as to 
minimize the current error. Such neuronal network ap¬ 
plications have received special attention because of their 
critical importance in deep learning (e.g. rat All in all, 


It does not take long to realize that algorithms for finding 
extreme of functions play a central role in several practical 
and theoretical circumstances. 



Figure 1: A scalar field defined on 9ft 2 , tak¬ 
ing value —1.000000000000019095836023... at 

(1.000000000000021982415887..., 1.000000000000021982415887...) 

, and -1.50000000022336310578907... at 

(1.999999999896590052728..., 1.999999999896590052728...). The 
respective mathematical expression is given in Equation Ob¬ 


serve that in this particular case the lower valley is also narrower, 
which makes it more difficult to be found. 

Because maximization of a function 'ip(x^y) can be im¬ 
mediately treated as the minimization of —^(x^y), for 
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simplicity’s sake we henceforth refer only to minimiza¬ 
tion of functions. Also, in this text we focus on scalar 
fields (i.e. functions taking scalar values) defined on two- 
dimensional domains. Scalar fields on higher dimensional 
domains can often be treated by using generalizations of 
methods developed for the 2-D situation. 

Minimization problems can be divided into two main 
groups with respect to the nature of their domain: (i) 
continuous ; or (ii) combinatorial. In the former case, the 
domain to be explored is continuous and one can move 
through connected trajectories, or paths, along it. This 
also allows eventual analytic performance approaches in¬ 
volving differential properties of the field. Finding the 
minimum of the scalar field in Figure [l] provides one ex¬ 
ample of continuous minimization. 

In the latter situation, namely combinatorial minimiza¬ 
tion, there can be discontinuities and gaps in the function 
domain, or it can be completely discrete and even there 
many not even be any mathematical values (or labels) as¬ 
sociated to each of its parts. As a consequence, it is not 
feasible to define searching trajectories along this type 
of domains. An example of combinatorial minimization 
is the traveling salesman problem, in which one has to 
visit each of N cities without repetition and accumulat¬ 
ing the smallest overall journey length. In this case, the 
domain constitutes of sequences of exactly N names of 
cities, without repetition. Though one could map the city 
names into respective mathematical values, these would 
be arbitrary. Other combinatorial minimization problems 
include packaging and circuit routing. 

In the present work, we focus on continuous minimiza¬ 
tion. 

Function minimization involves two distinct issues: (a) 
finding the minimum value V’mm of the given function 
'ip(x^y)] and (b) finding the point(s) (x m ,y m ) where this 
happens, i.e. = ^(^mjZ/m)- Observe that there may 

be more than one point where the function take identical 
overall minimum values - which accounts for our above 
mentioning of ‘point(s)’. Priority can be given to (a), (b), 
or both. 

Interestingly, the resolutions with which and 

{xmi Um) can be obtained are not necessarily interrelated. 
For instance, the overall minimum value can be very 
near to our estimation (x me ,7/ me ) of (x m ,y m ), but yet 
'0(x me ,7/ me ) can be substantially different of and 

vice-versa. So, it is important to carefully define before¬ 
hand the resolutions we expect for estimating both 
and (x m ,y m ). 

It may come as a surprise that it can be very difficult 
to find the overall, or global , extreme of generic functions. 
One of the main problem here are with functions contain¬ 
ing local extremes, such as that illustrated in Figure [l] 
Several searching approaches can be attracted to a shal¬ 


lower relative minimum and get trapped there. 

Other problems are implied by functions with discon¬ 
tinuities, steep variations, or with narrow valleys, valleys 
with very different widths, and so on, which can imply in 
search divergence and/or oscillations. Ultimately, there is 
no ultimate algorithm capable of ensuring the identifica¬ 
tion of global extreme of any generic function in a finite 
period of time. Indeed, because of the challenge and im¬ 
portance of this task, several methods have been proposed 
in the literature. 

From our discussion above, we can infer that suitable 
approaches to minimization depends substantially on the 
nature and structure of the function to be minimized, 
including the number and type of valleys, their scale, 
shape and proximity, etc. The interesting area of numer¬ 
ical analysis (e.g. [2]) includes studies of the performance 
of specific types of numerical minimization methods, and 
also of respective parameter configurations, given specific 
types of functions. 

In the present work, we aim at presenting, in a con¬ 
ceptual and introductory manner, some of the main min¬ 
imization concepts. We start by describing a simple ran¬ 
dom search approach, which allows us to infer the im¬ 
portance of the function domain dimensionality on the 
respectively implied computational cost. A zooming vari¬ 
ation of this random search is also outlined, allowing po¬ 
tential enhanced performance in some situations, allowed 
by the progressive reduction of the search domain. Gra¬ 
dient descent is then briefly described and discussed. To 
complement our presentation, simulated annealing-based 
minimization is also outlined. These four methods are il¬ 
lustrated with respect to a simple example involving local 
minima. 

2 Random Search 

The following presentation assumes the particular case in 
which the function to be minimized is a scalar field ^ de¬ 
fined on a two-dimensional domain A, yielding the func¬ 
tion 'ip(x^y). Extensions to higher dimensional domains 
can be derived. We also consider this domain A to be 
bound so that x min <x< x max and y min < y < y m ax • 

Let’s consider that the field 'ip(x^y) has its minimum 
value comprised within a circular region B centered at the 
point (x m , 2/m), indicated by an asterisk, corresponding to 
the overall minimum of and with a radius r, having with 
area a(B), as illustrated in Figure |2j It is also assumed 
that the function ^ is continuous and varies little within 
this region. 

Observe that the resolution (or precision) in which the 
position (x m ,2/ m ) can be obtained by this approach is 
directly related to r, in the sense that smaller values of r 
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tend to yield enhanced resolution in finding the position 
of the overall minimum. 

We henceforth adopt the fraction / relating the area of 
the green region and the total area as a(B) = fa(A ), so 
that the considered area containing the sought minimum 
can be understood as a fraction / of the total area a(A). 
Observe that 0 < / < 1. 



Pin(n) = 1 - P ou t(n) = 1 - (1 - fT (2) 

Figure [3] illustrates, in terms of the number of trials 
n and for several values of /, the probability Pi n (n) of 
having at least one of the points drawn in the random 
search falling inside the green region, where the overall 
minimum of the function 'ip(x^y) is located. 

This result illustrates the striking effect that / can have 
on the probability of finding the green region. For in¬ 
stance, let’s say that we are interested to have Pi n (n ) = 
0.8, corresponding to 80% of chance, after n trials, of 
reaching the region where the minimum is located. For 
/ = 0.0001, we need about n = 17000 trials, while for 
/ = 0.01 we can achieve the same probability with just 
n = 160 steps. At the same time, a vastly larger amount 
of trials is necessary for / = 0.000001. 


Figure 2: The function ^(x^y) to be minimized is constrained in 
a domain < x < Xmax and y m in < V < ymax with area 

a(A). The minimum of this function (at the asterisk) is assumed 
to be inside the green region with area a(B). It is also considered 
that the function psi is continuous and varies little inside the green 
region. By construction, any point inside this region is no further 
away than r from the point where the overall minimum is located 
(asterisk). 


A particularly simple method to search for the mini¬ 
mum, henceforth called random search (e-g. 0 ) , consists 
of randomly drawing points (xi,yi) uniformly distributed 
in the domain, evaluating the function at each of these 
successive trial points - yielding and keeping 

the minimum value obtained and the coordinates of the 
respective point. The method stops after i = n steps. The 
minimum of the function is approximated by the result¬ 
ing smallest value, that does not necessarily corresponds 
to the extreme, but to an approximation given the as¬ 
sumption that the function i/j is continuous and varies 
little within the green region (smoothness). 

It is possible to infer the probability of having at least 
one point inside region B after n trials. First, observe 
that the probability of uniformly drawing a point out¬ 
side the green region is P out = = 1 — /. If we 

henceforth consider that the probabilities of drawing each 
successive point are independent of the previous drawings, 
the probability of obtaining two consecutive points out¬ 
side the green region is equal to P° ut = (1 — /) . So, for 
n trials we obtain the probability of having none of the 
points inside the green region as 


Pont (n) = (!-/)" (1) 

The complementary probability, meaning that at least 
one of the drawn points fall inside the green region, can 
now be obtained as 



0 5000 10000 15000 20000 

n 

Figure 3: Probability Pi n {ri) of having at least one trial point inside 
the region where the overall minimum is, in terms of the number of 
trials n , for several values of the fraction /. Observe the striking 
difference implied by different values of / on the number of trials n 
required to achieve a given probability (ex. Pi n = 0.8). 


It is possible to derive, from Eq.[2j the following expres¬ 
sion for the number of steps required to reach the green 
region with probability P^ n after n trials: 


_ ^Pf?(l Pin ) 

log(l-f) 


(3) 


Figure [i] depicts n in terms of / = 10^, k = 

1,1.5, 2,..., 5, for Pi n = 0.8. 

As it can be observed in Figure [4j we again have that 
the number of trials increases steeply with / for a fixed 
Pi n . In other words, the fraction / plays a key role in in¬ 
fluencing the computational cost of finding the region con¬ 
taining the overall minimum. So, it is interesting to con¬ 
sider how the random search approach performs for func¬ 
tions with higher dimensional domains, let’s say $l N . If 
we assume that the regions A and B are hypercubes with 
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Figure 4: The number of trials required for having a probability 
of 0.8 of not missing the overall minimum region, for / = 10 fc , 
with k = 1.0,1.5, 2,..., 5. Observe the steep increase observed at 
/ = 0.001 for this specific case. Due to the logarithms in Eq. [3] 
this transition can be observed at another value of / when other 
sequences of / are considered. 


sides W and w, respectively, we have that a(A) = W N 
and cl(B) = w N , so that / = (j^) N • From Equation [ 3 J 
we find 


n(N) = 


logjl - P in ) 

iog{ 1 -(#)") 


(4) 


Assuming ^ = 0.1, implying relatively little resolu¬ 
tion, and Pi n = 0.8, we obtain n( 2) = 161; n( 3) = 1609; 
and n(10) = 16094377792. This result emphasizes the im¬ 
portance of the dimension of the function domain on the 
respectively implied computational cost. Problems that 
are relatively simple in lower dimensions can become un¬ 
treat able for larger dimensions. That is another reason 
why minimization has received particular attention. 


3 Zooming Random Search 

The previous random search method always sampled the 
whole region A , which implies in several of the trials 
missing the region H, therefore increasing computational 
overhead without contributing to the achieved resolution. 
This is especially undesirable because the required num¬ 
ber of steps for a given probability of finding the mini¬ 
mum increases steeply for smaller values of /. It would 
be interesting to improve this aspect by considering some 
adaptive scheme. 

Here, we resource to an approach based on the progres¬ 
sive reduction of the search area A , based on focusing on, 
or zooming into the region of interest, where the overall 
minimum is located. 

It is possible to progressively reduce the search area 
A, giving rise to a sequence of search areas A i: i = 
0,1, 2,... ,m, each centered at the current minimum 


value, which would yield successive improvements in the 
resolution. However, such an adaptive scheme needs to 
avoid the situation in which the sought region B results 
outside the decreasing region A , implying the overall min¬ 
imum to be missed. 

In order to devise some reasonable adaptive scheme, 
we need to have some statistical indication of the spacing 
between the points within the region A after a given num¬ 
ber of trials n. For instance, the average nearest neighbor 
distance between these points can be understood as be¬ 
ing related to the size of gaps in the spatial coverage. If 
we can estimate this distance at any number n of trials, 
we can use this information to derive a possible zooming 
methodology for random search that is unlikely (but by 
no means guaranteed) to miss the region B containing the 
overall minimum value of ^(x^y). 

In an homogeneous two-dimensional distribution of 
points with intensity (density) A, the distribution of near¬ 
est distances d between any point of the plane and any 
existing point can be obtained as 

p(d) = 2Xnd e~ X7Td , (5) 

with respective cumulative distribution (e.g. 0 ) 

P(D) = p{d < D) = 1 - e ~ XnD2 , (6) 

Observe that 


A = 


n 

a(A) 


(7) 


Strictly speaking, Equations [5] and [6] refers to a Poisson 
point process (e.g. si), but also hold for spherical contact 
distributions and can be considered as an approximation 
in other types of nearly uniform point distributions. The 
average nearest distance between pairs of points can be 
obtained from Equation [5] as 


<d> = 


1 

2a/A 


( 8 ) 


Figure |5] illustrates two different uniform point distribu¬ 
tions considering a(A) = 1 and n = 300 (a); and a (A) = 1 
and n = 3000 (b), and the respective circles (in orange) 
having as radius c = 5 (arbitrary value) times the value 
of the respective average nearest distances d. 

The zooming approach requires some scheme for choos¬ 
ing a smaller search region at each step i. A possible ap¬ 
proach is to impose a fixed probability Pf of keeping the 
overall minimum within the next search region. We have 
from Equation [6] that 


D = 


logp - P f ) 

\i r 


(9) 


For instance, in case we have n = 10 points inside a 
search region with area a A = 1, it follows that A = 10, 
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Figure 5: Two different uniformly random point distributions con¬ 
sidering at(A) = 1 and n = 300 (a); and a(A) = 1 and n = 3000 
(b), yielding A a = 300 and A^ = 3000, respectively, from which we 
obtain d a = 0.02886 and = 0.009128. The circles shown at the 
middle of each image have 5 d as respective radiuses, i.e. five times 
the respective average nearest neighbor distances. 

and for P(D) = 0.8 (i.e. 80% of probability of not missing 
the minimum), we have that D = 0.05123. Therefore, it 
is possible to pre-establish P/, fix the number of trials at 
each step as n, and then obtain D defining a new square 
window of side 2D centered at the coordinate where the 
current minimum was found. Starting in a region A, a 
possible respective algorithm would be 

1. Search the region A by performing n trials uniformly, 
and keep the minimal value so far found, as well 
as its respective coordinates (x me ,7/ me ); 

2. Define a new searching area A' as the square centered 
at (x me ,7/ me ) with side corresponding to 2D (given 
by Eq. [9J with A = ^), so that a(A') = 4 D 2 . Make 
A = A!, obtaining the new search domain. 

3. If the resolution has not been achieved yet, or if a 
maximum number of steps has not been reached, go 
to 1. 


It should be kept in mind that, as with the previous 
random search algorithm, it is not possible to ensure the 
overall minimum value will not be missed along the suc¬ 
cessive interactions above. 

The convergence of the method depends on the type 
of scalar field to be minimized, the size of the attraction 
basin around the overall minimum value not to be too 
small while being relatively separated from other relative 
valleys, among other constraints. So, this method is by 
no means expected to converge for generic scalar fields. 
In addition, observe that the equations adopted in the 
above method refer to point distributions in 5ft 2 , so that 
extensions to higher dimensional function domains require 
other specific respective equations to be considered. 

4 Gradient Descent 

Let x , y ) be a scalar field (i.e. a scalar function) on some 
domain A G 5ft 2 , so that (x,y) G A. In case the partial 
derivatives and ex i s ^ we can define the 

gradient of fi(x : y) at the point (x,y) as: 

-t dip(x,y) - d^(x,y) - 

**(*.!/) = —‘ + 3 ( 10 ) 

It is possible to extend the gradient to higher dimen¬ 
sion domains. Observe that the gradient is a vector 
field derived from yielding a vector at each point 

(x,y). These vectors have a particularly important prop¬ 
erty: they point to the direction of maximum variation 
of fi(x,y) at the point (x,y). Informally speaking, if we 
understand the scalar field as a topography (e.g. a moun¬ 
tain), the gradient will point to the direction of steepest 
ascent. 

As an example, consider the simple scalar field given 
as fi(x,y) = x 2 + y 2 , defining the paraboloid. The gra¬ 
dient of this scalar field can be immediately obtained as 
Vfi(x^y) = 2xi-\-2yj. So we have that, for this specific 
function, the gradient radially points outwards from the 
center of the coordinate system, which coincides with the 
direction of greatest variation of fi(x,y). 

This suggests that we can try to reach the minimum 
of this scalar field by stepwise movements contrary to the 
direction pointed by the gradient at each position, giving 
rise to the minimization method known as gradient de¬ 
scent (e.g. [5]). In the case of function maximization, the 
method becomes hill climbing). 

More specifically, we start at some domain point (2^, yfi) 
and move contrary to the gradient by a step of magni¬ 
tude A Vif(x,y), repeating this step as many times as 
needed to obtain a small variation of the estimated min¬ 
imum f if{x rne ,y rne ), or until a limiting number of steps 
is achieved. Observe that this method implies a discrete 
trajectory, corresponding to the sequence of coordinate 
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values obtained at discrete steps, to be defined along the 
function domain. Also, it is often interesting to consider 
steps equal to A V^(^, y)/\\Vi/j(x, y)\\, i.e. with magni¬ 
tude A independent of the gradient intensity. In this way, 
the trajectory can proceed faster in the less inclined por¬ 
tions of the function, and not too fast when it its more 
inclined. 

It is often interesting to have the step A to adapt dur¬ 
ing the descent, in the hope to increase resolution and/or 
reduce the computational cost. The identification of par¬ 
ticularly suitable adaptive resolution schemes is an inter¬ 
esting (and relatively difficult) problem related to the area 
of numeric analysis (e.g. 0). 

In this work we consider the simple, and relatively 
limited approach, in which we make A = a A, with 
0 < a < 1, after each set of m steps. It should be kept 
in mind that adaptive step schemes can strongly impact 
the accuracy of the results, and even imply divergence or 
oscillations. 

Because the gradient descent approach only considers 
the most immediate neighborhood (actually infinitesimal) 
around the current point before each successive step, it is 
called a local methodology. In addition, given that it al¬ 
ways takes the steepest descent, without considering other 
possible alternatives, it is sometimes called a 1 greedy ; ap¬ 
proach. One consequence of the latter characteristic is 
that the gradient descent will follow through a path pos¬ 
sibly leading to a local valley, and therefore miss the global 
minimum. 

The region around each local minimum so paths start¬ 
ing at any of its point will converge to the respective min¬ 
imum is often understood as a respective attraction basin. 
The two other methods previously discussed in this work, 
namely the random search and zooming random search, 
can also be understood to have a ‘greedy’ dynamics. 

Figure [6] shows level sets obtained for the considered 
scalar field in Figure [lj while the red and blue arrows 
indicate possible trajectories followed by gradient descent 
starting from respective points. Observe the existence of 
two respectively defined attraction basins, one of them, 
with minimum near (1,1), being a local minimum. 

An important issue associated with minimization by 
gradient descent regards the need to estimate the gradi¬ 
ent numerically, e.g. by considering some finite difference 
scheme with resolutions Ax and Ay. 

5 Simulated Annealing 

An interesting approach to minimization is based on an 
analogy with statistical physics concepts and phenomena, 
especially the metallurgic process of simulated annealing 

(e.g. 0El). 
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Figure 6: Contour plot (level sets) of the scalar field in Fig^ The 
two attraction basins can be easily identified. The trajectories in red 
(shallower valley) and blue (deeper valley) illustrate possible search¬ 
ing paths followed by gradient descent from the respective starting 
points. Observe that less straight paths would have been obtained 
in the case of elongated valleys, which could imply oscillations along 
the respective trajectories. 

In metallurgy, the material of interest (e.g. steel) is sub¬ 
jected to heating and controlled cooling, so as to mini¬ 
mize the energy implied by the interaction of the crys¬ 
tal domains (e.g. make these crystals smaller and more 
aligned/organized). Observe that, in case the cooling is 
not performed properly, the misalignment of crystals im¬ 
plies relatively large tensions to appear locally in the un¬ 
accomodated material structure, reducing its mechanical 
strength. 

An important characteristic of simulated annealing 
minimization is that it does not always take the steepest 
alternative, which would imply in ignoring other possi¬ 
bilities. Instead, simulated annealing also allows taking 
less steep moves, or even moves taking into higher func¬ 
tion values (‘energy’). However, this is performed with a 
probability that decreases with the ‘badness’ of the move, 
i.e. the relative increase in the variation of the function. 
In order to incorporate a temperature parameter, allow¬ 
ing the search to ‘cool down’, a probability distribution 
involving temperature as a parameter is considered. The 
basic idea is that, as the temperature is progressively low¬ 
ered, the search has the change of being ‘trapped’ into the 
global minimum attraction basin. 

By allowing the search to proceed to higher function 
values, the chance of missing the overall minimum is de¬ 
creased. In the unfeasible situation of an infinitely long 
cooling period of time, the global minimum could be 
reached. Therefore, the simulated annealing method can 
be understood as a non -‘greedy’ approach. This type of 
method is particularly interesting when we are more inter¬ 
ested in not missing the overall minimum than on achiev- 
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ing a fast execution time. 

Here, we focus on the Boltzmann (also known as Gibbs) 
distribution, given as 

1 £ i 

Pi = -e"** (ii) 

where Z is a normalizing constant (the partition func¬ 
tion ), T is the temperature, k is the Boltzmann constant, 
and Si is the energy associated to one of the possible states 
i = 1,2,. ..,7V. 

The partition function corresponds to: 

N 

z = J2 e ~^ ( 12 ) 

As we are interested only on the probability of energy 
variations S = — £*, it is possible to consider the ratio 

between the respective probability densities, which yields 
the Boltzmann factor, given as 


Pi+i 

These concepts have been adapted (e.g. 0 ®), through 
an analogy with the Metropolis-Hasting algorithm, to 
yield the following criterion often adopted in simulated 
annealing minimization: 


respective reduction of the search region. For instance, if 
the simulated annealing strategy is incorporated into the 
random search discussed previously here, allowing the 
function to increase sometimes would have little other 
effect than generating fluctuations. 

So, it is necessary to devise means to couple the pro¬ 
gressive reduction of larger function increases with the 
extension of the domain area being searched. Ideally, it 
would be expected that nearing the minimum value would 
be accompanied by approximation to the respective coor¬ 
dinates, and vice-versa. This type of behavior is favored 
by continuous, relatively smooth functions. 

It is possible to devise gradient descent methods incor¬ 
porating simulated annealing. For instance, a very simple 
and relatively limited possibility is to add perturbations 

5 to the gradient with some given probability /3. Some¬ 
times it can be interesting to have not only a tempera¬ 
ture reduction scheme, but also some means of reducing 
the magnitude of the perturbations along the searching 
procedure. 

6 A Case Example 

In order to illustrate the application of the methods de¬ 
scribed in this text, we consider the following scalar field: 


1 if 

e t if 

(14) 

Therefore, steps leading to a smaller value of the func¬ 
tion to be minimized are always taken, while steps im¬ 
plying relative increase are taken with a probability that 
penalizes large increases and that tends to decrease as T 
is reduced (i.e. the search is cooled down). 

Observe that computational implementations of this 
criterion can adopt the Monte Carlo approach to obtain 
the desired probability = Po- This involves 

obtaining a value v uniformly drawn in the interval [0,1] 
and taking the probability in case v < Pq. An impor¬ 
tant related aspect concerns how the temperature T is 
allowed to decrease as the search proceeds. This issue 
is not particularly simple, as it depends on the structure 
and properties of the function to be minimized. For sim¬ 
plicity’s sake, here we consider repeatedly reducing T by 
a ratio 0 < (3 < 1 after each sequence of m steps. 

Another important point regarding simulated 
annealing-based minimization concerns the fact that 
it makes little sense to take into account only the values 
of the function, while overlooking the relative proximity 
between successive coordinate points. In other words, it 
is of no use to progressively reduce variations of the func¬ 
tion values along the trials if this is not followed by some 


ip(x, y ) = -e~ - 1)2 ° + °^ 1)2 - ^ e - 16 [(^- 2 ) 2 +(y- 2 ) 2 ] ( 15 ) 

This function has two local minima, one approxi¬ 
mately equal to —1.00000000000001909583... observed at 
(1.00000000000002198241..., 1.00000000000002198241...), 
and another with value —1.5000000002233631057... 
at (1.99999999989659005..., 1.99999999989659005...). 

Therefore, it takes its global minimum value in the latter 
case. Figure [l] illustrates this function. 

First, we apply the random search method, starting 
with A defined as —3 < x < 5 and —3 < y < 5. Fig¬ 
ure [7] depicts the value of the estimated global minimum 
in terms of the execution steps. 

The estimated global minimum in this execution 
yielded the value —1.49271715..., which is relatively coarse 
approximation to The estimated coordinates of 

the minimum point were (x me = 2.00762427..., 7/ me = 
1.98431365...). The estimated minimum value was ob¬ 
tained at step 84268. 

Next, we consider the zooming random search of the 
same scalar field and adopting Pf = 0.99999 and n = 200. 
Figure [8] shows the progression of the successive search 
areas, starting at point (1.0,1.2). 

Figure [9] illustrates the estimated overall minimum val¬ 
ues in terms of the number of trials i for the zooming 
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Figure 7: Estimated minimum (vertical axis) in terms of the number 
of trials i obtained for one execution of the random search of the 
scalar field in Equation |15| in the search region A defined as —3 < 
x < 5 and —3 < y < 5. 
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x 

Figure 8: Successive search regions defined by the zooming ran¬ 
dom search method for the scalar field in Eq. |15| starting at point 
( 1 . 0 , 1 . 2 ). 


The estimated global minimum in this execution 
yielded the value —1.500000000223363105789..., which 
is very close to the expected global minimum value 
'ipmin • The estimated coordinates of the minimum 
point were ( x me = 1.999999999896239000..., y me = 
2.00000000084977846981...), which present a relatively 
larger shift from the expected values. The estimated min¬ 
imum value was reached after 2927 trials. This result il¬ 
lustrates the improved efficiency allowed by the zooming 
approach for some types of scalar fields. 

The application of gradient descent adopted numerical 
estimation of the gradient by using a first order finite dif¬ 
ference scheme with Ax = Ay = 0.0000000001. We fixed 
A = 0.009 (i.e. a = 1), and started at point (1.4,1.8). 
Figure [To| depicts the progression of the search. Observe 
that the convergence to the overall minimum was only 
allowed because the starting point was contained in the 
respective attraction basin. 
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random search considering the above mentioned assump¬ 
tions. 


Figure 10: Successive steps along the gradient descent of the scalar 
field in Eq. |15| starting at point (1.4,1.8). The reduction of the 
magnitude of the successive steps was implied by the decreasing 
gradient magnitude along the discrete trajectory. 



Figure 9: Estimated minimum (vertical axis) in terms of the number 
of trials i obtained for the zooming random search of the scalar field 
in Equation |15| adopting Pf = 0.99999 and n = 200, and starting 
at point (1.0,1.2). 


The values of the estimated overall minimum obtained 
in terms of the successive steps i are shown in Figure [TT| 
Our very simplistic approach to gradient descent incor¬ 
porating simulated annealing involved implementing per¬ 
turbations with fixed magnitudes \S\ = 3 and probability 
0.1 as the temperature, starting as T = 10000, was de¬ 
creased to 1/10 after each 100 steps. The starting point 
was (1.0, 0.8). Figure 12 shows the successive steps taken 
along the execution. 

Figure [13] illustrates the estimated overall minimum val¬ 
ues obtained along the search. Though in the initial stages 
the search briefly aimed at the local minimum, the pertur¬ 
bation scheme allowed the overall minimum to be reached 
and followed more smoothly as the temperature decreases. 




















Figure 11: Gradient descent estimation of the overall minimum 
value in terms of the number of steps i. 



X 


Figure 12: Trajectory of the gradient descent incorporating sim¬ 
ulated annealing method starting at point (1.0, 0.8). Observe the 
exchange between the two attraction basins. 



Figure 13: Obtained overall minimum values along the gradient 
descent incorporating a simplistic simulated annealing scheme. 

It should be observed that the evaluation and compar¬ 
ison of the errors obtained during minimization can often 
be more effectively performed in terms of relative errors 
defined as 


_ I /misestimated) - f m i n {real) | 

fminireal ) 

It would be an interesting exercise to obtain the relative 
errors for all above situations. 

7 Concluding Remarks 

The subject of function minimization is a particularly im¬ 
portant and interesting recurrent problem in science and 
technology. In this work, we developed some related con¬ 
cepts and ideas, starting with a simple random search ap¬ 
proach and the progressing to a respective zooming ver¬ 
sion. These two methods involved some concepts from 
probability theory (e.g. [7]) and stochastic geometry, in¬ 
dicating that their extension to higher dimensions can in¬ 
volve steeply increasing computational costs. Both these 
methods rely, at each step, only on the obtained values of 
the scalar field, not considering other information. 

A local approach, based on the estimation of the gra¬ 
dient at each of successive points, defining a respective 
discrete trajectory along the scalar field domain, was then 
presented. This method considers not only the minimum 
values obtained so far, but also the respective gradient, 
which drives the search. Though this approach often al¬ 
lows the valleys to be tracked in a more objective way 
than random search, it is subjected to possible oscilla¬ 
tions, divergence, and getting stuck at local minima. 

The principle of simulated annealing minimization was 
briefly introduced as a possible means for avoiding the 
chances of getting stuck in local minima. Its effective 
convergence, however, relies on careful choice of temper¬ 
ature cooling schemes, as well as progressive reduction of 
the perturbations magnitude. 

The relative characteristics of each of the four presented 
minimization methods, all of which assumed scalar fields 
defined on subsets of 5ft 2 , were illustrated and briefly com¬ 
pared assuming a relatively simple scalar field containing 
two valleys. The random search turned out to be the sim¬ 
plest but less effective scheme, while the zooming random 
search provided substantial improvements in the specifi¬ 
cally considered case. The gradient descent approach al¬ 
lowed faster convergence in this particular case, but relied 
critically on the choice of the start point being within the 
overall minimum attraction basin. The gradient descent 
incorporating simulated annealing allowed, in the case of 
the specific considered execution, the relative minimum 
to be avoided. These results should be considered only 
as a didactic illustration of some of the basic features of 
the adopted methods, and any more formal comparison 
should involve repeated executions and several distinct 
parametric configurations. 
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It is hoped that our presentation has also illustrated 
the critical dependence of the performance of any given 
minimization method on the parametric configurations, 
starting points, as well as several features of the scalar 
field to be minimized, including the distance between rel¬ 
ative minima, their respective heights, the shape and size 
of the attraction basins, among many other aspects. Here, 
we assumed that the scalar field was continuous and rela¬ 
tively smooth near the respective minima. Recall that no 
existing method can be guaranteed to reach (and some¬ 
times not even converge) the overall minimum in the case 
of generic functions. 

It should be kept in mind that many other interesting 
minimization methods exist, such as the elegant Nelder 
and Mead’s, simplex, or amoeba approach (e.g. 0 0). 
The present work constitutes only a brief conceptual in¬ 
troduction to some important aspects of an interesting 
and ever developing area. 
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Costa’s Didactic Texts - CDTs 


CDTs intend to be a halfway point between a 
formal scientific article and a dissemination text 
in the sense that they: (i) explain and illustrate 
concepts in a more informal, graphical and acces¬ 
sible way than the typical scientific article; and 
(ii) provide more in-depth mathematical develop¬ 
ments than a more traditional dissemination work. 

It is hoped that CDTs can also integrate new 
insights and analogies concerning the reported 
concepts and methods. We hope these character¬ 
istics will contribute to making CDTs interesting 
both to beginners as well as to more senior 
researchers. 

Each CDT focuses on a limited set of interrelated 
concepts. Though attempting to be relatively 
self-contained, CDTs also aim at being relatively 
short. Links to related material are provided in 
order to complement the covered subjects. 

Observe that CDTs, which come with absolutely 
no warranty, are non distributable and for non¬ 
commercial use only. 

The complete set of CDTs can be found 
at: https://www.researchgate.net/project/ 

Costas-Didactic-Texts-CDTs. 


10 





